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ABSTRACT 


The  Electronic  Systems  Division  (ESD)  has  been  stressing  the 
importance  of  Reliability  and  Maintainability  (r/m)  in  the  development 
and  acquisition  of  ground  electronic  systems/equipments.  This  report 
discusses  elements  of  ESD  R/M  programs.  The  development  of  qualitative 
and  quantitative  requirements  are  indicated.  Methods  and  criteria 
involved  in  the  evaluation  of  proposals  and  selection  of  contractors 
are  presented.  Finally,  contractor  monitoring  actions  performed  by  ESD 
activities  and  types  of  problems  observed  are  discussed. 


REVIEW  AND  APPROVAL 


This  technical  report  has  been  reviewed  and  is  approved 


i  .  _  Standards  Office 


iii 


CONTENTS 


SECTION 

I, 

INTRODUCTION 

1 

A. 

Progress 

1 

B. 

Organization 

1 

SECTION 

II, 

INITIATING  AN  R/M  PROGRAM 

2 

A. 

Statement  of  Requirements 

2 

1.  Quantitative  Requirements 

2 

2.  Modeling  Studies 

2 

3.  Some  Cost  Considerations 

3 

4.  Demonstration  Aspects 

3 

B. 

Qualitative  Requirements 

5 

1.  General  Statements 

5 

2.  Specific  R/M  Tasks 

5 

C. 

Source  Selection  and  Evaluation  Criteria 

6 

1.  Bidders'  Briefings 

6 

2.  Source  Selection  Board  Organization 

6 

3.  Proposal  Evaluation 

6 

4.  Rating  Process 

8 

5.  Contract  Negotiation 

8 

SECTION 

in,  r/m  program  monitoring 

10 

A. 

Monitoring  of  Contractor  Progress 

10 

B. 

Program  Problems  and  Actions 

10 

1.  Common  Problems 

10 

2.  Actions  Taken 

11 

SECTION 

IV, 

SUMMARY 

12 

A. 

Stating  Requirements 

12 

B. 

Evaluating  Proposals  and  Selecting  a  Contractor 

12 

C. 

Monitoring  of  Contractor  Progress 

12 

ILLUSTRATIONS 

TABLE  I. 

,  PROBABILITY  OF  ACCEPTANCE  TABLES 

13 

EXHIBIT 

1, 

TWENTY  R/M  TASKS  SUGGESTED  BY  MIL-R-27542  and  MIL- 
M-26512 

l4 

EXHIBIT 

2, 

EVALUATION  CRITERIA-  SPO  (Reliability) 

15 

EXHIBIT 

3, 

EVALUATION  CRITERIA-  SPO  (Maintainability) 

16 

SECTION  I 


I.  INTRODUCTION 


A.  Progress 

During  the  past  ten  years  several  significant  milestones  have 
heen  achieved  hy  government  and  industry  teams  with  respect  to  the  de¬ 
velopment  of  the  reliability  and  maintainability  (R/M)  engineering 
disciplines.  As  examples,  we  cite: 

Incorporation  of  quantitative  requirements  in  system  and 
equipment  specifications . 

.  Publication  of  a  family  of  reliability  specifications,  in¬ 
cluding  the  recent  MIL-R-27542A,  and  a  basic  maintainability 
specification,  MIL-M-26512 . 

.  Approved  demonstration  methodology  which  has  capitalized  on 
the  various  Epstein/Sobel  publications. 

.  Establishment  of  basic  R&D  programs  leading  to  improved  pre¬ 
diction  techniques  and  more  comprehensive  understanding  of 
physics  of  failure. 

and  finally, 

Development  of  improved  management  structures. 

B.  Organization 

The  establishment  of  contractor  r/m  organizations,  with  specific 
responsibilities,  lines  of  communication  and  authority,  has  an  analog 
within  the  Air  Force  Systems  Command  (AFSC) .  Each  Division  of  AFSC  has  a 
Staff  Office  specifically  charged  with  the  responsibility  for  R/M  and,  in 
addition,  focal  points  are  assigned  to  Systems  Program  Offices  (SPOs). 

Within  the  Electronic  Systems  Division  (ESD)  this  Staff  Office  is  assigned 
to  the  Technical  Requirements  and  Standards  Office  (EST),  which  in  turn 
is  assigned  to  Assistant  for  Staff  Support,  who  reports  directly  to  the 
ESD  Commander.  The  assignment  of  R/m  focal  points  to  ESD  SPOs  has  been 
accomplished  through  the  receipt  of  Air  Force  Institute  of  Technology  (AFIT) 
graduates  with  M. S.  degrees  in  System  Reliability. 

It  is  this  Staff-SPO  team  which  is  assigned  the  responsibility  for 
the  successful  accomplishment  of  ESD  R/M  programs.  This  team  is  supported 
with  limited  resources  from  the  Rome  Air  Development  Center  (RADC)  and  the 
MITRE  Corporation.  For  those  who  are  interested,  AFSCM  375-3*  System  Pro¬ 
gram  Office  Manual,  dated  15  June  1964,  and  published  by  the  AFSC,  describes 
a  typical  SPO  organization  and  methods  of  operation. 


SECTION  II 

II  INITIATING  AN  R/M  PROGRAM 

The  ESD  SPO-Staff  team  has  several  key  responsibilities  to  discharge 
before  a  contractor  can  commence  his  part  of  an  overall  R/M  program. 

These  include: 

Preparation  of  statement  of  requirements. 

Establishment  of  evaluation  criteria  for  R/M. 

.  Assist  in  evaluation  of  bidder1 s  proposals,  negotiation  and 
award  of  contract. 

A.  Statement  of  Requirements 

Responsibility  for  statement  of  requirements  entails  a: 

.  Specification  of  quantitative  values  for  R/M  with  associated 
demonstration  models. 

.  Preparation  of  a  statement  of  work  (SOW)  which  outlines  a 

series  of  tasks  or  work  elements  to  be  accomplished  by  a  con¬ 
tractor. 

1.  Quantitative  Requirements 

System  quantitative  values  at  ESD  are  usually  expressed  in 
terms  of  steady  state  or  point  statistical  availability.  This  situation 
arises,  since  the  majority  of  our  systems  are  continuous  demand  or  non¬ 
mission  orientated  with  respect  to  operational  requirements.  Systems  which 
have  specific  mission  times  associated  with  them,  such  as,  ICBM  and  air¬ 
craft,  are  more  appropriately  defined  in  terms  of  success  probability,  MTBF, 
etc. 


The  expression  of  an  availability  requirement  implies  the  joint 
control  and  consideration  of  interfaces  or  interactions  between  R/M  values 
at  the  major  hardware  levels  of  a  system.  In  terms  of  subcontracted  hard¬ 
ware,  a  system  contractor  will  have  to  impose  numerical  requirements  in 
specifications,  upon  completion  of  system  modeling  studies,  to  have  a  prob¬ 
ability  of  successfully  achieving  or  exceeding  system  requirements.  System 
hardware  design  strategy  must  reflect  consideration  and  incorporation  of 
malfunction  or  failure  detection  methods,  application  of  redundant  replace¬ 
ments  and,  as  necessary,  alternate  modes  of  operation  with  less  accuracy, 
built-in  test  equipment,  etc. 

2.  Modeling  Studies 

Total  design  strategy  for  hardware  must  consider  and  evaluate 
a  matrix  of  competing  and  interacting  characteristics  including  the  ven¬ 
erable  engineering  constraints  of  volume,  weight  and  cost.  While  it  is  true, 
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that  ESD  systems,  such  as  474l  (BMEWS),  465L  (SAC)  and  4l2L  (USAFE) ,  are 
not  unduely  hampered  by  volume  and  weight  because  of  their  operational 
environment,  it  is  also  true  that  there  is  not  an  infinite  amount  of  dol¬ 
lars  available  for  our  systems.  Therefore,  it  follows  that  a  solution  to 
our  system  availability  requirements  must  not  limit  itself  strictly  to 
the  addition  of  hardware  in  redundant  configuration  with  its  associated 
cost  penalties. 

3.  Some  Cost  Considerations 

Beyond  hardware  design  considerations  and  their  arrangement 
into  a  configuration,  there  are  other  methods  for  consideration  in  design¬ 
ing  systems  for  availability  that  must  be  cost  analyzed  and  compared  with 
competing  design  strategies.  For  example,  cost  vs.  the  advantages  of 
numerical  incremental  effects  on  availability  of  manning  a  number  of  sites 
vs.  the  design  strategy  of  increased  hardware  reliability.  The  variable  in 
question,  time-to-travel-to-sites  from  fixed  geographical  locations  (re¬ 
pair  depots)  is  one  factor  affecting  system  down-time  and  thus  availability. 

Recently,  this  type  of  problem  confronted  ESD  on  a  system  pro¬ 
gram.  There  existed  the  need  for  increasing  system  availability.  Several 
strategies  were  advanced  for  achieving  this  need.  One  strategy  involved 
additional  computer  hardware  for  redundancy.  Another  was  concerned  with 
site  manning.  The  original  system  design  had  three -of  four  sites  unmanned. 
An  analysis  indicated  increased  availability  comparable  to  redundant  hard¬ 
ware  by  selective  site  manning  at  less  cost.  We  mention  this  type  of  system 
analysis  with  the  intent  to  encourage  avoidance  of  habit  forming  application 
of  hardware  redundancy  as  being  the  only  solution  to  achieving  our  avail¬ 
ability  requirements. 

4.  Demonstration  Aspects 

When  numerical  values  for  system  availability  are  set  forth, 
the  problem  of  demonstrating  such  values  usually  reduces  to  a  separate  veri¬ 
fication  of  hardware  r/m  characteristics,  thus,  the  importance  of  numbers 
incorporated  into  hardware  specifications,  and  a  mathematical  combination 
of  results  to  determine  the  level  of  system  achievement.  We  know  of  only 
one  approach  to  the  verification  of  availability  as  a  statistic  without  re¬ 
sorting  to  separate  R/M  verifications.  While  the  statistical  modeling 
approach  suggested  by  Bailey  involves  sequential  techniques  and  the  exam¬ 
ination  of  paired  sample  values  of  operating  time-to-failure  and  down-time 
or  time-to-restore  operations,  it  still  provides  no  relief  to  basic  ESD 
problems  of  dollars  and  schedules. 

At  the  hardware  level,  our  MTBF  numbers  are  usually  in  the 
500-1,000  hour  interval.  To  obtain  sufficient  paired  values  for  decision¬ 
making,  requires  a  waiting-time  or  test  length  which  might  interact  un¬ 
favorably  with  schedule  and  dollar  constraints.  In  other  words,  the  key  to 


1.  J.  H.  Bailey,  Foundations  of  Sequential  Testing  and  Application  to 

Availability  of  Data  Processing  Systems  (IBM  Technical  Report  TR00.992)* 
March- 5,  19S3". 
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the  reported  approach  is  to  obtain  a  paired  value  sample  but  to  do  this  p.n 
the  face  of  high  MTBF’s  requires  increases  in  test  time  which  may  be  pro¬ 
hibitive  when  interfaced  with  other  program  considerations.  With  these 
thoughts  in  mind,  we  take  the  following  position,  at  this  time,  with  re¬ 
gard  to  system  and  hardware  demonstrations: 

.  Separate  hardware  reliability  demonstrations 
Separate  maintainability  demonstrations 

.  Statistical  combination  of  the  results  to  determine  system  values 

With  relatively  high  equipment  MTBFs,  we  have  been  applying  variations 
of  fixed  test  time  modeling  techniques  to  our  hardware  reliability  dem¬ 
onstration  problems.  Certain  of  our  hardware  specifications  have  required 
operation  within  performance  requirements,  under  specified  environmental 
conditions,  for  discrete  multiples  of  MTBF,  with  a  cited  maximum  allowable 
number  of  failures  (C) .  Inherent  in  this  type  of  approach,  as  illustrated 
by  Table  1,  are  statistical,  engineering  and  management  considerations. 
These  are  discussed  in  depth  in  ESDP  80-5  and  ESDP  80-8^. 

The  Cummulative  Poisson  expression,  which  is  the  statistical 
model  for  this  Table,  indicates  the  statistical  risks  or  probabilities  of 
accepting  hardware  having  MTBF  less  than  the  required  MTBF  as  C  -  values 
and  test  time  vary.  ESD  and  contractors,  upon  examining  such  an  array  of 
numbers,  can,  at  least,  numerically  recognize  the  other’s  risks.  This  in 
itself  should  increase  the  probability  for  more  satisfactory  demonstration 
agreements . 


Engineering,  as  in  any  demonstration  model,  must  define  hard¬ 
ware  failure,  cite  failure  counting  ground- rules,  specify  the  environmen¬ 
tal  spectrums  and  test  methods.  To  management,  the  fixed  test  time  model 
permits  more  accurate  scheduling  and  budgeting  than  the  sequential  model, 
for  example.  Finally,  for  all  key  personnel  assigned  to  a  program,  the 
model  is  easy  to  understand  and  administer  (which  is  probably  its  greatest 
advantage l ) 


Maintainability  demonstration  for  hardware  requires  a  specifi¬ 
cation  of  a  minimum  number  of  failure  simulations  and  logging  time-to-re- 
store  operation.  Fifty  or  sixty  representative  simulations  are  usually 
involved.  The  identification  of  what  should  be  simulated  is  influenced  by 
the  results  of  and  knowledge  gained  during  engineering  design  reviews  as 
well  as  statistical  predictions  of  reliability. 

The  statistical  combinations  of  hardware  results  to  achieve 
system  values  usually  follows  the  well  known  probability  models  and  ex¬ 
pressions  originally  postulated  for  reliability  predictions.  The  much 
over-worked  product  rule  has  found  new  applications. 


2.  These  two  ESD  documents,  Verification  of  Quantitative  Reliability  Re¬ 
liability  Requirements  and  Construction  and  Application  of  Probability 
of  Acceptance  Curves,  are  available  to  industry  as  part  of  ESD-TDR  64-6l6. 
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B.  Qualitative  Requirements 


1.  General  Statements 

SOW  requirements  for  R/m  find  background  information  in  the 
previously  referenced  MIL-R-275^2  and  MIL-M-26512  specifications.  Those 
in  ESD  who  apply  these  specifications  are  familiar  with  their  contents 
and  realize  the  need  for: 

.  Supplementary  instructions  relative  to  the  tasks  or  work 
elements  suggested  within  the  specifications. 

.  Selection  and  definition  of  tasks  per  program  and  the  avoid¬ 
ance  of  such  statements  as,  "Reliability  shall  be  in  accordance 
with  MIL-R-275U2m. 

.  Integration  of  individual  reliability  and  maintainability  work 
requirements  on  a  task  by  task  basis. 

2.  Specific  R/M  Tasks 

ESDP  80-2,  General  Requirements  for  a  Reliability  and  Main¬ 
tainability  Program  Plan  for  Electronic  Systems,  sets  forth  some  basic 
tasks  which  are  considered  to  be  applicable  to  all  our  system  and  equip¬ 
ment  programs.  These  tasks  cover  a  wide  variety  of  work  requirements 
ranging  from  mathematical  modeling  to  submission  of  reports.  We  view 
certain  of  these  tasks,  in  addition  to  modeling  and  demonstration,  to  be 
essential  to  the  achievement  and  maintenance  of  technical  control  for 
R/M.  As  examples: 

.  A  task  entitled  "Design  Reviews”  should  insure  that  such  basic 
reliability  engineering  considerations  as  conservative  applica¬ 
tion  of  piece  parts  and  techniques  for  minimization  of  environ¬ 
mental  stresses  influence  design  strategy.  A  design  review  task, 
as  defined  by  ESD,  includes  types  of  reviews,  schedules,  minutes, 
participation,  and  necessary  corrective  action  follow-up. 

A  task  entitled  "Corrective  Action  Management"  accomplishes  the 
need  to  insure  that  timely  and  effective  corrective  actions  occur 
on  problems  limiting  hardware  and  system  R/M. 

and  finally, 

A  task  entitled  "Predictions  and  Modeling"  provides  the  basis  for 
estimating  numerical  effects  of  changes  influencing  R/M,  track¬ 
ing  progress  toward  the  achievement  of  numerical  requirements, 
suggesting  design  improvement  areas,  and  for  necessary  re-apportion- 
ments  of  system  requirements  at  subsystem  and  hardware  levels. 

These  tasks  in  addition  to  others  selected  from  the  family  of  tasks  defined 
in  MIL-R-275^2  and  MIL-M-26512  (see  Exhibit  l)  are  listed  and  described  with¬ 
in  our  RFPs  and  IFBs.  These  tasks  when  scheduled,  defined  in-depth, 


integrated  within  the  overall  program  and  negotiated  with  a  procuring 
activity  or  SPO,  form  the  basis  of  a  contractor’s  R/M  program. 

C.  Source  Selection  and  Evaluation  Criteria 

1.  Bidder’s  Briefings 

We  recognize  that  on  occasion  industry  may  not  interpret  the 
requirements  in  the  manner  intended  by  ESD.  To  avoid  management  and  tech¬ 
nical  problems  at  a  later  date,  arrangements  are  made  for  a  Bidder’s 
Briefing,  usually  a  week  after  RFPs  are  issued  and  three  to  four  weeks  be¬ 
fore  formal  Proposals  are  to  be  submitted.  At  a  Briefing,  bidder’s  are 
invited  to  submit  in  writing  questions  on  any  item  within  an  REP  that  they 
feel  requires  clarification  or  interpretation.  The  questions  are  answered 
and  when  necessary  REP  contents  are  modified. 

2.  Source  Selection  Board  Organization 

As  stated  previously,  ESD  inserts  R/M  requirements  into  RFPs. 
Having  placed  requirements  into  these  documents,  we  now  want  to  be  certain 
that  "end  items”  (hardware,  software,  etc)  will  satisfy  our  needs.  As 
a  first  step  in  achieving  this  certainty,  Source  Selection  Boards  (SSBs) 
are  established  to  evaluate  proposals.  A  first  action  of  a  Convened  Board 
is  to  develop  specific  criteria  and  a  rating  plan  covering  weights,  factors, 
etc.,  to  be  used  in  evaluating  proposals.  These  criteria  are,  of  course, 
based  on  bid  package  requirements. 

Generally  speaking,  a  SSB  is  composed  of  three  or  four  Groups. 

Each  Group  is  assigned  the  responsibility  for  a  given  area  of  a  proposal. 

These  areas  include  such  large  groupings  as  management,  engineering,  tech¬ 
nical  requirements  and  logistics.  Each  Group  may,  in  turn,  consist  of 
several  Teams  which  have  responsibility  for  one  or  more  specialities  or 
factors.  For  example,  the  Technical  Requirements  Group  may  have  several 
Teams  responsible  for  such  factors  as  reliability,  maintainability,  quality 
assurance,  test,  configuration  management,  etc. 

Personnel  assigned  to  each  Group  and  Team  are  drawn  from  var¬ 
ious  governmental  activities  as  needed.  They  may  come  from  a  SPO,  research 
activities,  staff  elements,  or  even  other  AF  Commands  (Air  Defense,  SAC, 

TAC).  Every  effort  is  made  to  obtain  qualified  personnel  for  each  speciality. 
Usually,  The  MITRE  Corporation  will  act  as  technical  consultants  to  SSBs. 

All  SSB  members  are  instructed  in  matters  of  Board  conduct  and  discipline 
in  order  that  proper  and  objective  decisions  are  made  during  the  life  of  a 
SSB. 


3-  Proposal  Evaluation 

Proposals,  when  received,  are  studied,  analyzed  and  evaluated 
by  Teams  assigned  to  a  SSB.  This  activity  is  accomplished  in  depth  and 
proposals  are  rated,  not  against  each  other,  but  against  established  cri¬ 
teria.  Typical  R/M  criteria  are  presented  in  Exhibits  2  and  3- 
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If  a  proposal  contains  information  requiring  clarification, 
depending  on  the  type  of  procurement  involved,  the  involved  bidder  may 
be  contacted  to  clarify  his  proposal.  Any  clarifications  or  additional 
information  must  be  submitted  in  writing  to  a  SSB. 

Let's  consider  several  R/M  subfactors  presented  in  Exhibits 
2  and  3-  Paper  limitations  prevent  us  from  giving  an  in-depth  discussion 
of  each  subfactor  but  we  feel  their  illustrations  serve  to  illustrate  the 
general  thinking  of  a  SSB. 

.  Past  Performance  -  We  are  interested  in  knowing  what  other 
programs  a  bidder  has  undertaken  which  involved  comparable 
R/M  requirements  and  the  results  obtained.  RFPs  request  a 
bidder  to  report  on  this  subfactor.  We  are  in  the  process 
of  developing  a  procedure  to  enable  a  retrieval  of  existing 
reports  and  data  on  past  accomplishments  in  order  to  gain 
further  information  on  a  bidder’s  performance. 

.  Experience  -  The  work  experience  of  personnel  being  assigned 
to  a  program  is  of  interest  to  a  SSB.  While  names  denoting 
reputations  in  R/M  are  impressive  at  first  glance,  we  have 
noted  in  the  past  that  these  are  frequently  "paper”  assign¬ 
ments.  It  is  important  that  the  qualifications  and  experience 
of  people  assigned  to  a  position  in  a  line  R/M  organization 
be  set  forth. 

.  Organization  -  The  organizational  structure  of  a  bidder  must 
indicate  the  lines  of  authority,  communication  channels  and 
inter-departmental  relationships  which  affect  his  R/M  organ¬ 
ization.  We  cannot  expect  to  find  his  R/M  organization  re¬ 
porting  to  a  line  design  manager  and  then  be  assured  of  the 
effectiveness  of  a  proposed  design  review  task.  Organiza¬ 
tionally,  we  expect  the  R/M  activity  to  interface  with  other 
activities,  be  in  a  position  to  influence  design  strategy 
and  management  decisions.  Furthermore,  while  R/M  are  considered 
design  characteristics,  the  influence  of  a  R/M  organization 
must  continue  throughout  the  testing  and  production  phases  of 
a  program. 

.  Scheduling  -  Proper  scheduling  of  R/M  tasks  is  of  major  im¬ 
portance  in  the  successful  management  of  a  program.  For  example, 
design  reviews  must  be  accomplished  prior  to  release  of  hard¬ 
ware  drawings  to  manufacturing.  Considering  the  reliability 
demonstration  problem  confronting  ESD,  a  careful  integration  of 
the  demonstration  task  within  the  schedule  structure  is  certain 
to  be  carefully  evaluated.  By  the  way,  we  have  evaluated  sev¬ 
eral  proposals  which  contained  demonstration  tasks  which,  if 
adopted  as  outlined,  would  have  been  in  excess  of  anticipated 
contract  lengths! 


7 


Control  of  Subcontractors  and  Vendors  -  The  type  of  program 
affects  the  prime- subcontractor  relationship  and  influences 
the  management  techniques  invoked  to  assure  satisfaction  of 
overall  system  requirements.  Briefly,  we  expect  a  prime  con¬ 
tractor  to  have  a  management  system  which  examines  and  corrects, 
when  necessary,  subcontractor  facilities,  procedures,  organ¬ 
ization  and  program  controls.  A  prime  contractor  must  plan  to 
indoctrinate  subcontractors  into  overall  program  requirements 
and  be  prepared  to  maintain  active  management  surveillance  and 
provide  technical  support,  as  needed,  to  subcontractors. 

Analysis  of  System/Equipment  Problem  Areas  -  Virtually  all  ESD 
procurements  have  state-of-the-art  engineering  type  problems 
which  affect  r/m.  When  one  considers  the  complexity  of  oper¬ 
ations  and  functions  required  of  our  systems,  this  is  under¬ 
standable.  Since  these  problems  exist,  specific  solutions  or 
compensating  system  design  features,  which  lead  to  trade-offs, 
are  rigorously  evaluated.  Since  ESD  RFPs  state  the  require¬ 
ment  for  r/m  analyses  and  models,  we  expect  that  the  quantita¬ 
tive  effects  of  problems  will  be  assessed. 

.  Other  Program  Elements  -  MIL-R-27542  and  MIL-M-26512  suggest, 
as  we  have  stated,  a  wide  variety  of  work  elements  or  tasks 
(see  Exhibit  l) .  Not  all  of  these  may  be  inserted  into  a 
particular  RFP.  If  a  bidder  feels  that  certain  tasks  should 
be  added  to  those  in  an  RFP,  we  expect  to  see  some  justifica¬ 
tion  for  his  position  in  his  proposal. 

4.  Rating  Process 

All  factors  (reliability,  maintainability,  etc.)  and  subfactors 
(organizational  structures,  design  review  activity,  demonstration  model, 
etc.)  do  not  receive  the  same  weight  in  determining  a  total  score  for  a 
proposal.  While  no  specific  values  can  be  stated  here,  in  a  usual  SSB, 
r/m  factors  contribute  about  30  -  40$>  of  the  total  score  received  in  the 
Technical  Requirements  area. 

When  one  considers  the  type  of  complex  systems  involved  in  ESD 
procurements  and  the  many  competing  disciplines  or  factors  involved  in  the 
rating  process,  a  30  -  40$  weight  is  a  sizeable  contribution  to  the  total 
score. 


Furthermore,  it  is  also  possible  at  this  time  that  a  bidder  may 
achieve  an  overall  acceptable  proposal  but  still  be  relatively  weak  in  R/M 
or  some  other  discipline. 

5.  Contract  Negotiation 

It  is  anticipated  that  at  the  time  of  selection  of  a  contractor 
all  program  requirements,  including  R/M,  would  have  been  satisfied.  Un¬ 
fortunately,  while  most  problems  will  have  been  overcome,  some  may  still 
remain.  Applying  the  negotiated  contract  approach  covered  by  AS  PR  III-8, 
further  discussions  and  definitions  of  a  R/M  program  can  be  undertaken. 
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The  RFP  and  contractor’s  proposed  r/m  Program  Plan  are 
utilized  in  these  discussions.  Meetings  are  held,  the  first  shortly 
after  contractor  selection  is  announced,  and  are  continued  as  necessary. 
Each  R/m  task  is  discussed  to  assure  that  full  understanding  exists  be¬ 
tween  ESD  and  contractor  as  to  requirements  and  actions  needed.  If  nec¬ 
essary,  changes  are  made  in  the  Proposed  Plan.  Minutes  of  all  meetings 
are  kept  and  are  fully  documented. 

ESD  is  cognizant  of  the  need  for  economy  in  system  acquis¬ 
itions  and  is  receptive  to  suggested  r/m  modifications.  Trade-offs  of 
R  vs  M,  R  vs  cost,  etc.,  will  be  considered.  We  do  try  to  stay  within 
budgetary  limitations  on  costs  and  will  consider  redirection  or  reemphasis 
of  certain  r/m  program  tasks,  if  proper  quantitative  analysis  is  submitted 
in  support  of  suggestions. 

Finally,  an  important  consideration  or  constraint  that  we  all 
must  adapt  to  in  planning  our  programs  is  the  time  compression  problem. 

In  order  to  meet  the  schedules  required  by  governmental  committments,  it 
is  sometimes  necessary  to  start  work  on  a  contract  almost  coincident  with 
contractor  selection.  In  these  cases  ESD  may  authorize  implementation  of 
the  R/M  Program  on  a  specific  task  basis  with  those  tasks  or  portions  of 
tasks  on  which  agreement  is  not  complete  being  delayed  pending  final  resol¬ 
ution.  For  example,  selection  of  suppliers  and  vendors  may  proceed  at 
once  but  additional  study  may  be  needed  before  approval  of  the  Failure  Re¬ 
porting  System  is  granted. 
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SECTION  III 

III.  r/m  program  monitoring 

A.  Monitoring  of  Contractor  Progress 

Following  initial  guidance  meetings,  upon  Award  of  Contract,  and 
approval  of  contractor  program  plan  by  procuring  activity,  verification  of 
progress  against  the  requirements  of  the  approved  plan  is  performed  by  the 
previously  referenced  SPO-Staff  team.  ESDP  80-7,  Monitoring  of  Contractors 
Reliability  and  Maintainability  Programs,  serves  as  a  general  guide  for 
accomplishing  this  management  responsibility. 

Monitoring  encompasses  a  wide  variety  of  activities  including: 

.  Participation  in  preliminary  and  critical  engineering  design 
reviews. 

.  Review,  evaluation,  comments,  and  approval  of  prediction,  pro¬ 
gress,  demonstration  and  failure  summary  reports. 

.  Review  of  progress  at  meetings  conducted  at  contractor  facil¬ 
ities  and  necessary  re-direction. 

.  Witnessing  formal  R/M  demonstrations. 

B.  Program  Problems  and  Actions 

1.  Common  Problems 

During  the  past  several  years,  our  monitoring  activities  have 
caused  corrective  actions  with  regard  to  technical  and  management  aspects 
of  our  programs.  Some  common  major  problems  which  are  described  in  ESDP 
80-4  include: 

Failure  of  contractors  to  accomplish  timely  design  reviews. 

.  Inaccurate  modeling  practices. 

.  Absence  of  successful  interface  between  contractor  R/M  and  de¬ 
sign  organizations.  We  still  recall  (unhappily)  that  we  brought 
about  the  introduction  of  reliability  and  engineering  managers 
during  a  monitoring  visit  to  a  large  company  working  on  one  of 
our  more  critical  projects. 

Absence  of  well  defined  corrective  action  management  procedures. 
On  one  program,  field  failure  data  flow  was  short-circuited  and 
design  engineering  was  unaware  of  several  major  problems  being 
reported  to  us  by  the  contractor's  R/m  organization. 
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2.  Actions  Taken 


These  problems  are  mentioned  to  illustrate  the  need  for  con¬ 
tinued  management  interest  in  the  r/m  disciplines.  To  help  sustain  such 
interest,  we  have  taken  the  following  diverse  actions  on  two  recent  pro¬ 
grams  undertaken  by  ESD: 

On  one  contract,  reliability  was  introduced  as  an  incentive 
factor.  The  opportunity  for  increased  contractor  profit  as 
a  result  of  achieving  a  specified  quantitative  objective  was 
provided. 

On  another  contract,  payment  of  fee  is  being  apportioned  over 
a  two  year  period.  The  quantitative  requirements  and  contractual 
structure  requires  the  achievement  of  specific  system  objectives 
during  a  first  year  of  operation  following  site  installation 
and  activation.  These  objectives  are  more  severe  for  the  second 
year.  By  providing  contractual  structure  which  influences  re¬ 
sulting  profit,  we  hope  to  sustain  management  interest  in  R/M. 
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SECTION  IV 


IV.  SUMMARY 

Tnis  report  has  described  elements  of  the  ESD  R/M  Program.  We  state 
that  our  responsibilities  can  be  grouped  into  three  general  categories: 

A.  Stating  Requirements 

Quantitative  expressions  and  contractor  program  requirements 
are  incorporated  into  our  RFPs  and  IFBs. 

B.  Evaluating  Proposals  and  Selecting  a  Contractor 

Bidder  R/M  responses  to  our  requirements  are  factors  in  the 
overall  contractor  selection  process.  The  weight  given  to  R/M  is  not  con¬ 
stant  but  varies  with  such  factors  as  nature  of  a  procurement,  severity 
of  quantitative  requirements,  etc.  A  cursory  examination  of  an  RFP  and 
IFB  reveals  several  competing  disciplines  which  influence  ultimate  contract 
award. 


C.  Monitoring  of  Contractor  Progress 

This  activity  generally  is  performed  by  a  SPO-Staff  team.  The 
competency  of  this  team  has  greatly  increased  through  the  assignment  to 
ESD  SPOs  of  AFIT  graduates  with  an  M. S.  degree  in  System  Reliability. 
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TABLE  NO  1 

PROBABILITY  OF  ACCEPTANCE  TABLES 


o 

CD 

-P 

O 

II 

o 

c  =  1 

C  =  2 

1/4 

0.02 

0.10 

0.26 

0.0003 

0.0024 

0.01 

1/2 

o.l4 

0.42 

0.70 

0.02 

0.10 

0.26 

l 

0.37 

0.74 

0.87 

o.i4 

0.42 

0.70 

3/2 

0.51 

0.85 

0.97 

0.27 

0.62 

0.85 

2 

0.61 

0.92 

0.37 

0.74 

0.92 

NOTE  1:  6,/e  =  "TRUE"  MTBF/CONTRACTUAL  MTBF 

*c  c 

NOTE  2:  FIRST  PROBABILITY  NUMBER  FOR  THE  CASE  WHEN 

TEST  TIME  =  CONTRACTUAL  MTBF;  SECOND  FOR  CASE 
WHEN  TEST  TIME  =  TWO  TIMES  CONTRACTUAL  MTBF. 
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EXHIBIT  1 


Twenty  R/M  Tasks  Suggested  By 
MEL-R-275^2  and  MIL-M-26512 

1.  Develop  and  Submit  r/m  Plan 

2.  Develop,  Modify  r/m  Mathematical  Model 
3*  Perform  Predictions  of  R/M 

4.  Apportion  R/M  Requirements  Into  Subcontracted  Equipment  Specifications 

5.  Organize  and  Maintain  a  Failure/Maintenance  Data  Collection  System 
(Data  Collection,  Processing,  Reduction  and  Feedback) 

6.  Maintain  a  Corrective  Action  System 

7.  Perform  R/M  Design  Reviews  and  Recommend  Appropriate  Design  Strategy 

8.  Review  ECPs  and  Non-ECPs  For  R/M  Effects 

9.  Submit  Monthly,  Quarterly,  and  Final  Reports 

10.  Coordinate  With  Quality  Assurance,  Human  Factors  and  Logistical 
Support  Personnel 

11.  Conduct  R/M  Indoctrination  and  Training 

12.  Perform  Failure  Analyses 

13-  Propose  and  Conduct  Part  Improvement  Programs 

14.  Perform  Maintenance  Analysis 

15.  Demonstrate  R/M  Requirements 

16.  Perform  Trade-Off  Evaluations  and  Alternate  Solutions  to  R/M 
Problems 

17*  Manage  and  Control  Subcontractor  Effort 

l8.  Participate  in  Progress  Reviews  with  Procuring  Activity 

19-  Review  Processes  and  Procedures  for  Fabrication  and  Assembly 

20.  Recommend  Changes  to  Operating  Procedures,  Maintenance  Manuals, 

Test  Equipment  and  Spares  Requirements. 
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Principle 
SOW  para  no. 


EXHIBIT  2 


Date 


EVALUATION  CRITERIA  -  _ SPO 

TECHNICAL  REQUIREMENTS  &  STANDARDS  GROUP  (AREA) 
R/M/QC  TEAM  (ITEM)  Reliability 

(Factor  -  10  points) 


♦ 


Sub- 

Factor 

Weights 

TT 

(  ) 


Adjusted. 

Raw 

Sub -Factor 

Score 

Score 

~n 

1.  Understanding  of  the  Impact  and  Importance  of 
Reliability  Program. 

~TT~ 

(  )  2.  Program  Plan  Development  -  General  considerations  (  ) 

and  tasks  below: 

a.  Allocation  of  reliability  requirements  to  each 
level  (subassembly,  component,  equipment)  at  each  stage 
(design,  development,  production). 

b.  Design  reviews  -  plans  for  at  least  three  design 
reviews  (part,  electrical,  mechanical)  scheduled.  General 
approach  including  planning,  staffing,  and  action  modes  re¬ 
sulting  from  reviews. 

c.  Reliability  Predictions  -  Methods  used  to  make 
initial  predictions  based  on  data  or  approved  techniques. 
Approach  used  to  update  initial  predictions  based  on  de¬ 
sign  and  test  results. 

d.  Test  Program  -  Development  of  test  plans  at  each 
level  and  stage  and  the  engineering  approach  used  to  meet 
contractual  requirements  in  an  economical  manner.  Method 
proposed  for  collection,  analysis,  and  feedback  of  test 
results. 

e.  Test  Procedures  -  Use  of  reasonable  test  procedures 
to  prove  achievement  of  required  reliability.  Method  of 
establishing  confidence  and  risk  factors  and  remain  within 
program  schedule. 

f.  Reporting  Procedure  -  Planning  for  completeness, 
details  to  be  covered.  Establishment  of  problem  list  and 
corrective  action  set  up. 

g.  Establishment  of  mileposts  and  monitoring  check  points. 

h.  Subcontractor  control  -  Methods  to  be  used  in  con¬ 
tracting,  establishing  of  requirements,  control  of  work. 

i.  Training  and  Indoctrination  -  Planning  for  necessary 
training  and  indoctrination  of  all  company  personnel. 


(  )  (  )  3*  Reliability  Organization  and  Staffing  -  Is  the  reli-  (  ) 

ability  group  in  a  position  to  exert  its  influence  on  the 
program  (on  a  par  with  Engineering  or  Production)?  Is  the 
organization  staffed  with  sufficient  men  to  do  the  assigned 
task?  Is  the  experience  adequate  to  assure  full  compliance 
with  schedule? 


TOTAL  FACTOR  SCORE 
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EXHIBIT  3 


Principle 

SOW  para  no. 

EXHIBIT  3  Date 

EVALUATION  CRITERIA  -  SPO 

TECHNICAL  REQUIREMENTS  &  STANDARDS  GROUP  (AREA) 


r/m/QC  TEAM  (ITEM)  Maintainability 

(Factor  -  10  points) 

Sub- 

*  Factor  Raw 

Weights  Score 

tt  ~rr 

Adjusted 

Sub -Factor 
Score 

1.  Understanding  M  requirements  on  this  program  in-  (  ) 

eluding  where,  when  and  how  the  M  requirements  will  be 

(  )  (  ) 

established  for  this  system. 

2.  Program  Plan  development  -  General  Coordination  and  (  ) 

Tasks  below: 

a.  Allocations  of  M  requirements  to  each  level  (sub- 
assembly,  component  equipment)  at  each  stage  (design,  devel¬ 
opment  production) . 

b.  Design  Reviews  -  Plans  for  at  least  three  (3)  de¬ 
sign  reviews  (part,  electrical,  mechanical)  scheduled. 

General  approach  including  planning,  staffing,  and  action 
modes  resulting  from  reviews. 

c.  Maintainability  Predictions  -  Methods  used  to 
make  initial  predictions  based  on  data  or  approved  tech¬ 
niques.  Approach  used  to  up-date  initial  predictions 
based  on  design  and  test  results. 

d.  Test  Program  -  Development  of  test  plans  at  each 
level  and  stage  and  the  engineering  approach  used.  Method 
proposed  for  collection,  analysis  and  feedback  of  test 
results . 

e.  Test  Procedures  -  Use  of  reasonable  test  procedures 
to  prove  achievement  of  required  M.  Method  of  establish¬ 
ing  confidence  and  risk  factors  and  remain  within  program 
schedule. 

f.  Reporting  Procedure  -  Planning  for  completeness,  de¬ 
tails  to  be  covered.  Establishment  of  Problem  List  and  correc¬ 
tive  action  set  up. 

g.  Establishment  of  mileposts  and  monitoring  check  points. 

h.  Subcontractor  control  -  methods  to  be  used  in  con¬ 
tracting,  establishing  of  requirements,  control  of  work. 

(  )  (  ) 

• 

3-  M  Organization  and  Staffing  -  Is  the  M  group  in  a  pos-  (  ) 

ition  to  exert  its  influence  on  the  program  (on  a  par  with 

Engineering  or  production)?  Is  the  organization  staffed 
with  sufficient  men  to  do  the  assigned  task?  Is  the  ex¬ 
perience  adequate  to  assure  full  compliance  with  schedule? 

(  )  (  ) 

4.  Understanding  of  special  maintainability  requirements  (  ) 

and  the  integration  of  SPO  system  needs  into  the  overall 
system. 

TOTAL  FACTOR  SCORE 
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